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' ABSTRACT 

^ Levels of Use (LdO) intervieirs can' he used Lor 

fom'ative evaluation purposes in an interaediate education agency. 
Progress and services for training teache^ provided to districts are 
evaluated^ LOO intervievs give prograa staff unique infbcsaticn for 
iiprqving prograas. Prograa iaproi»aent is .aided by LOO interviews 
because Infornation is provided th%t prograa staff cannot obtain on 
their ovn. The infornation corroborates vheth'er prograas are 
proceeding according to plan; Evaluation ax\d prograa staffs develop a 
stronger working relationship and .both gain .an understanding of the ' 
prograa/ sell defined relevant criteria is the goal of the process.^ 
Survey Instruaents do not !re veal the saae kind of inforaation. The 
interview technique allows coaiunication and effectively identifies 
how teachers are using the innovation. A health/ of for native 
evaluation infornation is provided by the interviews xn the areas of 
student characteristics; iiaterials, objectives, testing, and 
classrooB aanageaent.* the infornation proved useful in identifying 
areas of need and allowed project staff to aake decisions for prograa 
iaproveaent.xlt is especially valuable in its application to the 
purposes of foraative evaluation by practitioners responsible for 
effective change in schools. The aajor liaitation cited was the cost 
to execute valid evaluation. CDVB) 



Bbproductions supplied by EDBS are the best tha^ can be. aade * 
* froB the original docuaent. . * 
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Levels of Use Interviews: A. Successful Formative Evaluation Tool 

Alan L.-'Roecks 
C ' * John H. Andrews' , 

• • - 
Evaluation Services 

Education Service Centec^ Region 20 

San Autonio, Texas 

Since* t?e passage of the Elementary and Secondary Education Act of 1^65, 
evaluation of educational programs often has relied on the^ experimental model. 
Answers were sought to the question "Dpes ^the program work or not?" All too 
frequently, studies resulted in "no significant. findings" and programs were 
judged failures^despite experimental evidence suggesting that the progrs^ 
did/ indeed, produce desireable results. One explanation for this discourag^ 
ing finding could be the failure of proper program implementation. .Evaluating', 
such prdgrams runs the risk of appraising ^"non-events" (Charters & Jones, 
1973)^ Research from the extensive Rand study on Title I ESEA pTagrams (Berman 
& McLaughlin, 1978) revealed that the program in many districts had not taken 

place, suggesting that failure of proper implementation may be more widespread 

j'' * ' - 

than previcJUsly thought. Freeman C1977) contends that many "no effecti' evalu- 

- \ 

atlons. can be explained by the failure of pro]g^ progra6i implementation: 

There may be programs that offer significant solutions to .defects in 
the human condition that have been passeA over because evaluation 

y failed^ to show an impact. Yet, in fact, the real reason for lack of 
impact was that the program was never implemented fully well, or at 

' ' all. Cp. 27) 

Some researchers, including Mushkin (1973) and Ross^., Freeman ^nd Wright 
(1979r see the problem due to faulty research procedures. Comparative. 



1 , 

le definition of evaluation used for this paper Is taken from the Phi Delta 
Kappa's National Study Coomittee on Evaluation: "Educational evaluation is the 
process of delineating, gbtaining, and providing useful informal^ion for Judging 
decision alternatives." (Stuff lebeam, Foley, Gephart, Cuba, Hammond, Merriman, 
and Provus, 1971, p. 40>. 



( 



•educational .studies may be based bn instructional models not setisitlve to the ' 

actual eni^iroiment. The dominance *of the scientific method limits serious 

consideration 'of alternatj|pe research paradigms. Variables gj^at can beYcon- 

' trolled, are included in.experi^ntal studies; other relevant variables, may be 

excluded. Writing a decade earlier, Stufflebeam et al (1971)' wisely advised 

that evaluation methodology must not be limited to tha experimental model: 

...Perhaps the greatest challenge facing the evaluator is overcoming 
the idea that evaluation methodology is identical to research method- 
ology. Equating' them forces cjertain constraints inimical to the 
purposes qf /evaluation and makes it , impossible to meet certain of \he ^ 
needs served by good ^evaluat ion. Cp. 22) ^ ^ 

Elaborating, the authors caution that conventional experimental design may not 
be appropriate in situations where the efficiency of the overall process, in- 
cluding desired outcomes, is studied: 

Perhaps the most damagii^ assertion about the application of ^conventional 
experimental design to evaluation situations is that it conflicts with 
the principle that evaluation should facilitate the continuous improve- 
ment of a program. Experimental designs prevent rather than promote 
changes in the treatments... 

It is probably unrealistic to expect directors of innovative projects to 
accept^hese conditions, because they obviously cannot cbnstrain a treat- 
ment to its original, undoubtedly imperfect form, just to insure internally 
valid end-of-year data. .. .concepts of evaluation are needed which could 
stimulat;e, not stiffle, dynamic defvelopment of^programs. (p. 26) ^ 

Qualitative Methodology 

Viable alternatives to the experimental model recently have been developed. 
Qualitative methodology has* gained increasing support ^rom evaluation investi- ^ 

itatofs CGebhardt, 1979). Because they provide insight into the fundamental 

i « 

processes of education, qualitative methodologies emphasize getting "closer 
to tbe data,** developing an understanding of the observed and describing the 
reallfcy of the situation (Rist, 1977). Comtaon methodologies include partici- 
pant Qr \non-participant observation and comprehensive interviews. 



Qualitative inquiry can be independent ©r* complimentary of qualitative "st 

Qualitative procedures document the need for quantitative studies. Making 

sure proper program implementation has occurred, detergiined from. ^qualitative ' 

inquiry, *is often necessary before x:arrying out a more quantitatively-oriented 

impact study. Information from qualitative studjPBS is useful in addre'^ssing 

the formative evaluation question "How cah this program.be improved?*! ' 

...If 6he agency is interested in information on its effectiveness 
in delivering service3, the evaluation can stu<iy the process of 
f program implementation and fi^d out the extent to which the program 
is producing the quantity, quality, and coverage of services that were 
expected. (Weiss, 1972, p. 75). , ^ / 

The flexibility of qualitative studies allows formative evaluation of 
educational training programs, where findings can be used for the purpose of 
program improvement. Summative evaluation, which involves quantitative pro- 
cedures, isi harder to carry out due to the "inherent difficulty in defining 
how training programs are implemented. Educational training programs often ^ 
do not follow a prescribed curriculum;, the goa^s of the program may be im/' 
plemerited differently across organizational levels (Anderson & Ball,S-978, ♦ 
Chapter 10). This paper describes'how|^^litative'methodolj)gyf levels of 
Use aoU), was applied successfully 'in the formative evaluation of a teacher 

training program. ^ ' ^ . ' " 

- _ « 

Levels of Use Interviews 

V 

. In a comprehensive review df-.^urrictilum and instruction impl^ent;ation, 
Fullan and Pomfret 'C197^Vevealed, that^a mijority of research studies centered 
on the. fidelity of implementation*. Fidelity, which Compares actual and in-^ 
.^^^^^yjijende use for the innovation, was researched in two distinct W^ys^-focusJLng 

eithir^Mi organizational change or specific curricular innovations. One tech- 
nique. identified by fullan and Pomfret for measuring curriculum implementation 



was Levels of Use (LoU) of the Innovation. The relationship of LoU to the ^ 
.concept bf fidelity of implementation is illustrated below. 

i . ■ • X ^ - , ' 

I Fi^lity of Implementation 

♦ Specific Cwtricular Organizational Change 

Innovations 

^/ - . 

CMeasured by) 

Leve^ of Use . * ' . 



Conceptually, LoU is pajx of the Cqncerns Based Adoption Model (CBAM) . CBAM 
accounts f o'^l^lementation jiising two approaches. One approach focuses on change 
in the innovation itself, the second identifies two. critical dimensions for de- 
scribing change flrom the indivj|(ual perspective. These dimensions are Stages of 
Concern about the Innovation CHall & Rutherford, 1976) and Levels of Use of the 
Innovation <Hall, Loucks, Rutherford, & Newlove, 1975). A series o£ longitudinal 
and^ cross-sectional studies have verified the existence of- seven Stages of Con- 
cern and eight Lev^fs^ of U)5e (See Appendix A). loU has received limited uSe as 
a curricular evaluatfon tool with emphasis placed on summative evaluation. 
Okpalobi (1979) demonstrated* the effectiveness of a reading staff- development 
program by researching the overall and eight component ^LoU ratings for 27 ' 

treatment and 21 control teachers. With a few exceptions with some , ambiguity 
at the higher end of the soale. Hall and Loucks (1?77) and Reidy and ilord' (1979) 
discovered a positive relationship between the LoU and student achievement. Hal\ 
and Loucks also found that teachers in their first year of use do not implement 

well and correspondingly da not have as high LoU^ rating. Other facts about 
LoU include: 

# Reliability. Jnterater reliability ranges from •87 to .96 for 
• ^ the overall' LoU rating. 

XoU .was^ developed by educators for educators. 



Validity. Ethnographic studies of all day teacher activity reveal 
I 



for 
. for 



a correlation of .98 between iaterview ratings and ratings from 
observation. 

# Overall implementation. LoU can determine -the degree of implemen- 
tation for ^individuals exposed to. treatment. 

To the best of the authors' knowledge,- LoU interviews have not been used 
formative evaluation purposes. This paper documents how LoU can be used 
this purpiDse in aji intermediate e^ducation agency. '! 
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Region 20 

The state of Texas Is divided ;lnto twenty regions served byt intermediate - 
^education service agencies. Established as part of the public education system 
of Texas in 1967, the service agencies offer school districts and teachers an 
opportunity .to receive specialized serviced thattnorraally would be beyon^the 
•reach,of the average si^ed school system. By participating in programs with 
the intermediate agencies, school systems take advantage of cost-saving bene- 
fits and program financing andj planning which. result from cooperative efforts. 
They also realize additional funding, and services that are available from state 
and federal services for regional programs of this type. Since Texas Service 
Centers are not in a regulatory position for the districts they serve « student 
achievement data generally is not available for "evaluatfion of Center programs. 
Evaluation at Region 20 has a dual thrust. First, technical , assistance 

» * 

is provided to districts. EValtffltion workshops are provided to district staff - 
on topics including how Levels of .Use can document whether or not the program 
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Is In place. Second, programs -^nd services for training teachers provided 

to districts are evaluated, '{enter staff represent evaluation "clients*" 

r ^ , ' " 

Evaluation focuses od how teachers implement Center progreims. LoU is one 

of several evaluation techniques employed. 

• ' " \ 

•The READS Ptogram ^ ^. 

READS was one of about 30 curricul^r programs hoiisedK at^Region 20 during * 
the 1978-79 school year. READS consisted of a sequence of student objectives 
and pre-posttests for grades 1-8. Teachers used these materials Hlong with 
a record keeping system to individualize reading instruction.. Funded fromf 
Tltl^e IV-C, ESEA monies, READs was was in its third (Ph^se III) 'and final year 
of operation when evaluate^. The teacher sample reported in this paper con- 
sisted of teachers trained during the previous two years q£ the program. 

The evaluation- f^us was both summative and forpiative. Objectives whose' 

.Intent have been quantified describe program direction. The audience for' 
♦ 

summative ^evaluation, where a judgement about program worth is made, were the 
funding. agency and Region 20 administration. The sximmative objective for 

READS required that at iSfAst 60%- of the teachers trained during the two previoi 

' * ' 2 ' . 

years be implementers. 



By May 30. 197.9, 60% of the teachers in Phase I and Phase II schools 
ajid 40% of the teachers in Phase III schools will be' implementing 
READS with all or a section of thein studenjC population. Accomplish- . 
ment will be shown by a representative sample of teachers judged im- 
• . plementer^ (LoU. Ill, IV-A, IV-B, V) . ' * . 

* 

The priihary users of evaluation* information are program staff. Th^ir - 
needs are met by evaluation information which address immediate, short term 



2 , 
The major product of the evaluation officje's first y^ar (1977-781 was to 
develop measurable objectives fcg^ach Region 20 progr^. (See Drezek, Este^ 
Roecks, & Andrews, 1980.) A decision to use LoU interviews /or measuring^ 
curricular implementation had not been* made when the objectives* for I?£ADS''* - 
were' submitted to the funding agency^ * Subse^uentJ^y , objectives were modif^ted* 
so as to be measured by LoU. 



needs. The formative evaluation oBjective for READS reflects, this interi;:: 

^ By June 30, 1979*, a. revised edition of the English version of READS' 
vin be^ developed by project sta^f . Revisions 'will be based,, in part, 
itifo^rmation 'from l*oU interviews with project teachers. *Acc6mplish- 
mentr of this, objective ^i^ be ^evidenced by a panel of three reading 
curriculum' experts selected by the ESC-20 evaluation component judging 
th€u revi&;ton to be meaningful and^ substantial. 



sraT cc 



Limitations 



READS' wafe one^of several complicated programs at Region 20 during the late 
1970s. Evaluating such an involved program proved difficult and time consuming. 
Moreovir, the evaluation repre^er\ted the first attempt at using LoU as a foi 
atlve evaluation tool. For these reasons, the authors were hesitant to use 
information from. READS for this paper .v'^ Despite these concerns, the evaluation 
of READS Vest shows how LoU interviews can be used in an applied setting. 
Evaluation findings collected were some of ^the most diverse and comprehensive 
available, ' 

The next section outlines 'the procedures currently used to carry out LoU 
interviews to evaluate -14 curricular programs. These procedures are similar 
to the ones used for READS. Problems encountered in' the pilot tun^ith READS 



are illustrative of difficulties which may be encountered by others using LoU 
for the first time. They include: 



^ The original sampling 
~ scheme had to be disr 
carded. About one- 
half of the teachers 

\ 

from the previous year 
could not be located. 
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• The liat of participants was not.' ccnplctc. The actual number of teach- 
. ers trained was not known. 

• Criteria for implementation were not piloted. Five criteri^were 
originany_idsiltified. One of the criteria was showri later tV^e 

, critical for iypleaentation. . . ' ' 

• Insufficient' information was collected for certain criteria. Especially 

f 

difficult was collecting information for non-users. Subseqaent data 
analyses was difficult.. 

• Interviews were done by five evaluators, although '70%, of the interviews 

♦ 

were collected by the same two evaluators preparing the final report. 

• Given the abovl difficulties, the evaluation cost was nearly twice as 



much a3 budgeted. 



/ 



' K 
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LoU PROCEDURES 



This section illustrates, current procedures for 'carrying out an evaluati£)n 

stiudy using LoU, The five steps involved in th^is procedure .are illustratec^by 

.exampjei' from READS/ Figure 1, presentfed at this time as an advanced organizer, 

gives the relative amount of evaluation time needed to carry "out each step of 

the process. About two-thirda of an evalua tor's a^ailable"^ time is divided ' * 

evenly between conducting interviews and analysing data. Another one-^ifth is 

f 

spent selecting and 9ontacting the sample. The remainder of this section 
describes procedures associated with each step-. 



Figure 1 * . , 

How an Evaluator's Available 
Time is Used 




Between 75Z Und 80% of .aft *evaluator 's time is available for direct evaluation 
.work as documented by time and effort record?. Remaining time is spent attend- 
ing staff meetings, professional development activities and carrying Out 
institutional responsibilities. • ' 




* ' ^ Identifyirig Implement a tioli Criteria 
' - A key role for^ evaluation. Is identifying the, goals aad objectives of a 
prqgraiw OWlsV, 1978). Agreeing pn^wfjat should ^e the criter:^^ f^r implemen- 
: tatiou'is a- time-^ensumlng ^ind QfJtetf frustrating proce3s. ' - 



Securing agreement requires 
strong working relationship 
between evaluation diid 
prqgram staff. . 




In ouAettingj-^^his* relationship is often present Stk to the emphasis on for- * 
-mative evaluation .where evaluation and program staffs work closely together. 

(^e characteristic unique to 'intermediate edu^iation agencies mak^fi negotiating' 

^ . . , , • *^ 

, the criteria diffijcul;. ' ^ , * 

The program developer and pro- 
itator/ y^TX gr^m jEacilitator are ^ten the^ 

same person, titled^ the progr^ 
maliager. ^ , 





.ERIC 



'/ 



-10- 



The* dual ro|le8 a;re in conflict. * As program developer, the program manager 
wants to •s^t the criterjp Strictly^ So that the description^of the program can 
be put in its' bjpt light. However^ in their role as change facilitator^ the 
same progr^yn^ manager wants ^o^ have very' genera^ criteria to increase the •prob- 

aU>iltty that ^ higher percentage of teachers will be £mprementers. 

^ ^ ' ' . ' 4. 

Implementation criteria are piloted during the first year of 4 program ♦ 

When 'determining tentative criteria, evaluation and program staff must decide 

what the innovatiiot^ "looks like" in the ^lassroom^ * They mu^t respond to such' 

queSTlons as: ' 

If teacher Smith participated in the program and la using the materials 
provided, and if teacher Jones does not participate at all, what would 
' . * be happening to teacher Smit^i's. cla^s that wouldn't be happening to' 
teacher Jones' class? ♦ _ . 

^ ' ^ • 

The criteria are then piloted on a, few teachers. Eace to face ijiterviews 
are carried out irresently with the possibility of using* telephone interviews 
being^ explored. Pilot interviews reveal teachers who!tuse programs in u^u^al 
*ways or who are borderline with re^tpect to one or more' of the criteria. In 



reviewing these cases evaluation and program staff have r^al data, not hypo- 
t^hetical , situations, which can be used tp better define the boum^aries of the 
program. From this, tjfie criteria can be refined apd fiinalized. New criteria 

may ^so be identif il^STr ^^^^jj^^ 

^ A clear definition of the innoyatio^ is essential in obtaining qualitative 
brmation ujseful to program staff, irrespective of the data collectioti instru- 
nt emt>loye"d. Our experienced suggested two types' of implementsation criteria 
could be identified; co incidentally, the same conclusion was drawn by researchers 



year of operation centers *on developing materials and ironine out 
Ln the -ins true tional system. Teachers may^not be trained until the 



4 \ 
The first 

' wrinldes in the -ins true tiona! system. Teachers may^not be trained until the 
program/has been operationalized six months or longer. Implementation 'is 
seen as the second and third year priority. Criteria identified during the 
first year remains essentially the same for succeeding years. ^ ^ 



of the University oL Texas Research and Development Center for^ Teacher educa^ 
, tion (See Hall & Loucks, 1978) • v These criteria are calle<i ''critical" and 
. "r^lat^Td." .Teachers must meet critical criteria in order to be implementing 
the program. Critical criteria represent primary program, goals identified 
after the program is operational. Related y&riteri^, which also describes' in- 
dividual implementation, are &een by the program manager as important but not 
essentia^ for 'implementation. Teachers who are users meet xhe impletientation 
criteria and most related criteria.. The 'finalized criteria for READS are 
• given .belowf • ^ ♦ ' ^ ' ^ 

Critical Criteria: , • 

t 1. The teachers must be using the student objectives containeL 
in tJtia' program. They might skip a few objectives or u^e^them 
in a slightly different order But, in general, they should 
follow ^the prescribed sequence. J 

Related Criteria: ' * ^' 

1. The^eacher must identif^ the< reading level (skills) of each 
individual child. ' ^ 
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The students must be grouped by^skill levels (one student 
can be considered a group) . «* 

The teacher must Be testing for^skill mastery by objective. 
The mastery level can be established by the teacher. 
The teacher must have *a beginning at organizing teaching 
resources to'go with the objectives."" will be a retrieval 

system such as card file or file folder. • 
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* » 
Identifying the Sample and Schedulihg Intervtews 
• * * J. 

. The sampling design depends on p'rogram' size th^w many districts, campuses, 

^ and teachers are served) and comp;e3city (tim^ of implementation is proportionate 

to complexity). The number of participant Interviews is proportionate to 
^.roject level of funding. Sampling is sttatified ran,doaly by campus. About 
•twenty teachers, are ihterviey^d per project. ' The relatively small sample size ' 

mean,, that data is'aggr^ted fof all teachers and is not anil^zed by d^strictV 

or campus level. / , 

^ Identifying the teacher sample can be a frustrating task. Teacher' turnover 

# » 

can be as high as 50%. This problem is acute when trying to locate previous 
participants in the program. Finding teachers who have changed campuses can 
also be perplexing. The sample drai^'mist be increased ^02 to 50% o^ required 
size. Replacement must be taken into account. 




V 

^^'A "S^econd problem. is that the list of 
participants, maintained by program 
staff, may be inaccurate. Nothing 
is more perplexing than calling a 
teacher who says "I am not involved 
^in the program and haven't seen the 
. program staff for ov%r one ^earl*' 




Scheduling interviews in our setting is an involved process. Time seems ^ 
^ ^ to fly by,- Because the service, center does not have ♦ 

V 

» 

regulatory, authority over districts ,^ permission ^rom 
district personnel must be obtained prior to conducting 
the interviews. Contact Is made with the central >off ice, * 
usually %rLth thfe supervisor, followed by -tt.call to the campus principal. . 
Teacher^ are^ then contacte4- If they cannot be r^ach^d at the initial phone 
call they are asked tg return the call. If possible, interviews are sche'duled 
during thev^onf-erence period. When Interviews are scheduled more than a week 
in advance, a reminder postcard is^sent^. Interviews are schedulejd at the con* 
venlence of the teagher^ ^ Th is does not alwavS coincide with what is convenient 
for evaluation staff* resulting in a loss of tiiie. \ - 

y Conducting the Interview * ^ . 



On the^ay of the inte^^view, the evaluator first goes to- the principal's 
office to stat^that he or she i$ at the school. In some c«^s the- principal 
may request the. evaluator to check into the office before leaVing the school. 
The interviewer then goes to the teacher's classroom "or other .desiiznated site. 
The first part of the interview' focuses on the^criteria for the program. For 
each criteria, the interviewer attempts to l^dentify how well a teacher is 
meeting the criteria. Oncfe the inf-ormation on the criteria is colleated the 
interviewer proceeds to use. the standard .LoU format.' (See Appendix'/) 
The interview is usally completed wit^iin thirty minutes'. 

At the conclusion of the interview, additional interview fnfonnation may 
be collected. This information spans aspects of the program not covered by 
implementation criteria, such as quality of materials or the provision of' 
technical assistance. Sdj^pleoental information also has been gathered success- 
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1 ^ 
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fully^by otb«-o«^ (Klenke and 'Narrows, 1980) • 

When«v«r possible, travel is coordinated 
so that several teachers in the same or nearby ' 



campuses are Interviewed. Coordinating this 



activity to be cost effective has 
proved difficult. 




Data Analysis 



All interviews/ are transcribed. Essential--fpr quality^ tapes are a top-of- 
the-liivB^'tape recorder and a secretary familiar' with' educational programs • 

and their related jargon is a definite 
asset. Transcribing is done'^for two 
reasons. First, it is cost e|f«ctlve* 
A person can read faster than they can 
hear or write. The' cost of transcribing 
is^ off set by eyaluat'or time ^a^ed, 
es^Teclally -when is summary of * twenty ipter- 
views, each with five''^ criteria, must be 
prepared. Quotes are Included in evalua- 
tion summaries to enhance authenticity. 
Time is saved if the evdluator is not 
..required tp review the tape and transcribe information.- Second, transcripts 
can be revieyed ana the accuracy of data verified,' The transcrlijts themselves 
are. a good management tool, Evaluating programs using LoU interviews represent^ 
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a nev activity for many Region 20 evaluation 'staff. Reviewing transcripts 
Vlth a knowledgeable colleague can be a beneficial learning experience lead- 
ing to a higher quality final report/ 

Interview information -is analyzed in a two step process. The first step 
is to determine what percentage of the teachers*^ are implementing based on 
wliether or not they m^et critical' criteria. The sunmaative objective called 
for 60Z of the teacheni to be implementers' CLevels III, IV-A, IV-B, and V).^ 
The fipdings shown byjTable 1 reveal that 65Z were implementing with*45Z of 
the teachers being at the lowest implementation level, Level III, mechanical . 



Tab|:e 1 

Lef6l of Use Rating of READS ^Teachers 
CN « 20) 



Level 



Percentage of Teachers 



0 - 


Non use 




35Z * - 




Orientation 




OZ 


^ II - 


Preparation 






III - 


Mechanical 




45Z • 1^ * 


IVA - 


Routine 




15Z 


^ IVB - 


Refinement 


• 


OZ 


V - 


Integration 




•5Z 


VI - 


Renewal 







The second step is to analyze interview information according to both 
critical and related criteria^ HoW each teacher implemented the innovatatlon 
according to criteria is recorded. Similar patterns or dominant configura-'* 
tions emerge (Hall^ Zigmari, & Hord, 'l979). 'Analyzing data by configuration 
presentl^^s not done in our setting,, although tutufe research efforts. 



could profit \>y examining <;onfigtirations. The relatively few teachers 
interyiewedv limited the number .of potential dominant, configurations , 
however . * 



The best infqnaation was obtai-aed by analyzing 
across criteria. Staff found this informapion most 
relevant to immediate program needs. One of 
the related criteria, for example, r^quir^d 
teacjiers to group students by ability level. 
All teachers^met this critei»ia,' Four 



0 



variations were id^tified. The most ^ 
popular grouping methods were completely , 
Individualized C^OZ) and a combination 
of large and small. groip techniques C30Z) • Figure 2 is a schematic depicting 
how READS data could analyzed either by criteria (across) or configurations 
Cdown), In conclusion, data analyzed across criteria instead of by configura- 
tioti best meets the unique curritular iii5)rovement deman^^of program staff. 




Figure 2. 



Inplementation for Iteads Teachers* 

A ' 

Teacher Identification Number 



Grouping^ , 1 2 3.4 11 12.13 14 . / ; , 19 20 



Critical Crl ftU 
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1. Identify KMllag Uv«i 



****** ^^^^^^^^^^^^^^^^^ 

c2* Grouping . ^ 

Completely individualized Cone ta^one) i^^^i^it 

Begin all students at one level or 
one of' two levels, but then b^gin 

grouping students based on'their ^ " ^ 

performance r it it 

Combination e£ large and small 

\ , group techniques. * * it it it 

************ ^*******\*****i***^^^^^^^^^^^^^^^^ 

3. TMtiag for Skli; Muctry . . • .4 

**** * / * m' 
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DOMINANT CONFIGURATIONS 
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DIStUoHlN AND IMPLICATIONS 

One of the few studies available on- program ^plementation' measured by 
liu is similar to READS. This evaluation studies the reading component of an^ ' 
Individually Guided Instructio9 Program (IGE) in the Austin lodep^endent " 
School District. A general c<(mparison teveals- READS was r»pt as well imple- 
mented as. IGE. Thicty-five percent of READS teachers were non-usdf-s, compared 
to- only 202 for IGE. ' More significant, most READS 'teachers were still 
struggling to get' the prograa off the ground CLoU III), Most IGE teachers, 
like teachers from other studies CHall & Loucks^977; Loucks, Hall, Rutherford, 
Newlove & George, 1976; Reidy & Hord, 1979) were implementing Ut the routine. 
l€Vel (LoU LV-A). The somewhat regressed implementation shown by READS par- 
ticipants can be|^xplained by\he complexity of' instructional design and " 
certain incorrect assumptions about how implementation proceeds. 

The assumption supporting many Center programs like READslwas that most 
teachers trained would be implementing aftei^one year. This wff no t valid. 
Subsequent .experience with other Region 20 programs demonstrated that 40 to 
60X of teachers were lapWnting after one year and up to 80% were implementing 
after two. The implementation process, stated Loucks et al (1976) takes • 
considerable time. • • • ' ' 



...our research* documerits the fact that implementing. . . takes time. 

It appears that teachers and professors, when implementing innovations, 

grow developaentally in their concerns and use of the innovation. 
They are not simply non-users of an innovation one day and sophisticated 
users the next. (p. 12) 
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Programs built on the. assumption that the implementation takes lust one 
year were, by design, ineffectual. Such programs diS not account for the 
• large number of previous ye^rs teachers wh^ were not impLement eri . As pro- 
grams continued, more .teachers needed to be serviced—usually with the same ' 
or a i^educ^d level of program resources. 

Some Years Later * 



First Year 






The long-term result was . that programs were less ef f icienf..-^nci^'-teachers did 

not receive the- support /eq'uired. Fewer teachers than expected became im- 

plementers; those who did implemeftt, did so at a lower level. Such was the 
case with READS. " ^ * 

Simpler innovations are easier to implement than complex ones. Facil- 



itators caij install them quicker and teachers can understand them readily. 
Simplicity is the key, especially wh^n One remembers' that the agency supplying 
the s«ed money wants to see^ near-immediate results. The management system 
linderlying READS was a complicated one, with similar systems frequently re- 
quiring computer asslstanii for imp-lementation (Baker, 1978 i Roecks, 1979b) 
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Instructlonally, many of the same systems-are not sound (rL^Ics, 1979a). 
One goal of manag^ent systems like READS is to save^ teacher time, -For READS 
an excessive amount of teacher time was required, as illustrated from teacher 
comments: 



I finished cha rting , well, 
the student profiXe charts. 
I finished that last week 
and here it is December and 
I have three (RMDS) clatees. 
If I would havJ'had ,five, ^ 
still would be working on 
that (User). 

We feel very strongly that 
it is too complicate, too 
much paper work, and it ' 
takes* away from actual class- 
room instruction (Non-user) . 




...here it is December* * 
^and I still have' three more 
ILIADS classes. 




r 



Strengths and Weaknesses 
Region 20 program staff contend that LoU interviews provide some of the 
best curricular evaluation information available. They believe LoU interviews 

give them unique information for imj>rov-* 
ing their programs. Staff a^ disap- 
pointed if the evaluation does not employ 
LoU interviews. They are no^ longer 
content with survey infdrmation or 
checklists. 




ERIC 



-21- 




AtV^Mst four reasons can_be cited why LoU interview information is 

useful for prograa improvement. First, information is provided that program 

staff cannot obtain on their own. Region 20 evaluation staff can secure j 

higher quality information due to their' 

i 

organizational independence from the pro- 
gram. Participants are reluctant- to 
"l^ll it lik^ it is" to those they work 
with. Moreover, confidentiality of infor- 
mation is guaranteed by the policy of our 
evaluation office. Second, LoU information 
corrobora^tes whether oz no^'the proi^^jSyp is 
proceeding according to p^an. The 
fidelity of implementation is shown by 
the degree to which LoU findings corresp&nd to critical and related criteria 
CSee Figure^l). Third, the process of having to develop specific criteria for 
the program has potential side affects. Evaluation and program staffs develop - 
a s.tronget working relationship and both gain an understanding of the program. 
^Tfie goal of the process, well defined relevant cr^eria, ultimately benefits 

the classroom teacher state Hall and Loucks (1^8): 

* / 

'...it appears, that when the developer is jiot clear in describing the 
innovations or change facilitators do not communicate effectively, 
* users are apt to implement an unusual configuration or at iBast de- 
scribe the innovation in different ways. On the other hand, if the 
implementatioo procesrs is well designed and articulated, and is 
consistent with the developer's model, the semantic and operation 
confusion is not so apt to occur. CP'/30-31) 

• ^ > ' 

Fourth, the k±nd of information resulting from LoU interviews^ generally cari- 

not be obtained from survey instruments. Questionnaire data represents one 

way communication where teachers are aske^^ to respond to program and evaluation « 
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3taffs' preconceptions of how teachers are using the program. This "Catch 22" 
phenonema can be summarized as follows: ' . 

*1. Questionnaires are sent out to determine what the teachers are 
, doing with- the innovation. 

r 

2.^ The only way the questionnaire^ will tollect this information is 
'^"^ if the questions asked are relevant to what the- teachers are si^p- ' 
posed t^o be doin^. . ' 
3* There is no way to ensure that the questions are relevant yitbout 
already knowing what the teachers a^e doing. 
^ 4. fif you alr«£jfj^ know what the tfeachers a're doing, there is no point 
^ in sending out the questionnaires - 
Interviews are most effective, in correctly identifying how teachers are 
using the innovation. Vrtiire the interviewers have a predetermined list of 
criteria, they are free to ^o "into other 
aspects or issues the teachers themselves 
raise. Unanticipated out comes ^^^^>kich do 
not direct^iy relate to Impiiamentation 
criteria, provide some of the mc^irt 
useful evaluation dat^« New criteria 
may emerge. Variations occur- in both 
the users and the innovation 'itself . 



f ' Boy! We 
^^sure got more than 
^"^^^^ asfe :ed for! ! ! 




Although evaluation in our setting is objective based, -important side effects 
can be identified 6y using techniques such as LoU. Examining program effects, 
irrespective of goals )ha^ been advocated ty Michael Scriven (1974), a pioneer 
in the area of goal*free evaluation. 
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Illustrative finding's Include: 

Student Population Served: ^ ' 

• READS was used almost exclusively with remedial or special 
education students. • ^ 

4 Some special education teachers were unaware that READS could 
be used in small group settings. 
Resource Materials: . t 

/• Some middle school' teachers have problems finding materials 

that their remedial students would not see as being "baby 
^ books;" ; % ^ 

• Teacl^ers frequently did not ha^e access to the materials listed 
for each objective. ' 

• Most teachers wanted the ^EADS cards laminated. p 
READS Objectives: ... 

• About half the middle school teachers believed objectives were^ 
too specific. - 

■- < 

• Some teachers perceived^a conflict between READ^ objectives 
and state and district guidelines. 

Testing and Class Management 

e Some teachers believed the pretest placed student^ at tto high 

a reading level. r ^ 

« Several teachers found 'pretesting required months to complete. 
\\ • Posttesting by objectives was sometimes not done because teachers 

^ needed more than one set of the posttests and :reproducttd.on corfts 

were high! , 

• '^me teacher3 were having trouble aatlaging bdi^ioral probil^ems 
in small group settings.^ . ^ - . 




The biggest drafiback to LoU interviews' is the cost arssociated with securing 
information useful for program modification. Carrying out the interviews and 
determining the overall LolT^s easy to accomplish. This task is inexpensive 

relative to the cost of giving form- 
ative information. The cost to carry 
oi^t an evaluation ^tudy using LoU' 
for* 20 teache|^ is given by Table 2. 
About three person weeks, represen- 
ting between 50? to 60% of the total 
cost, are required to complete the 
study. Transcribing^osts .are $20 
to $25 per tape. 

- - The cost g^en by Table 2 assumes jpersonnel ar^^ trained in applying LoU , ' 
as a formative evaluation tool. Evaluator^ must undergo a three day training 
session* in order to be considered for certification. Additional time, review- 
ing pre-rated dipes, is required for certifi- 
Cation. Learning .how to compile LoU d^ta 
for the purpose of formative evaluation' 

is a time consuming activity.' - . ( ^V^ahilM ^e are Certified 






Table 2 



lift 



Cost to Carry Out an Evaluation Stydy •Uslilg LoU for 20 Teachers^ 





Cost 

\ : 


Estimate 


t * * 

bxpiBuo X lu r e 

' [ r 


L 

iLow 


< 

High 


Personnel (about three person weeks) ^ 

« 


\ 

$ I 950 " 


$ l-,250 


Transcribing 20 tapes^ 


! 400 


500 


Overhead (travel, tape recorder, tapes, 
office equipment, supplies)^ 

—J 


'350' 


550 


Total 


$ 1,700 


$ 2,300 



Interview information is used ftfr both summatlve and formative purposes. If 
only summatlve inf^onaation is' required^^ (overall Lett), the cost is 30 to 40 
percent lower* 




Persomftl should fte certifiejd in Levels of Use, Cost iacl^des loss of three 
workdays plus cost of tralallng and travel expenses. These costs are not in- 
cluded- ' 



c* I 
This represents the amunt paid to an outside business. We do our o\rp. tvan^:,^ 

scribing^ resulting iVhigher quality at a slightly lover cost. Initia]|||^est- 
ment inclindes.a transcribing unit (§850,00). and typewriter .($800,00), W^' 



3 

As Region 20 serves 50 schcfols in fourteen counties much of the overhead cost 
is travel. Tape recorders j (Lanie?: dictaphones) cost $250, OO', tapes $4,00.. 



1 




ERIC 



'.26-: 



SuBsnary 



This stud^ presented an evaluation of a reading program whose intent 
8. teachers ovfer a three year period. A sample of 



was to train 



-20 teachers was Interviewed using :he Levels ^f Use technique. Criteria 
for successful implementation were identified". Findings revealed 65% of' , 
the teachers implementing at varloiis' levels, wit^he'degree of implemen- 
tation being lower 1:han for similar studies, the. regressed implantation 
was accounted for by the complexity of instructional design and the in- 
correct assumptiott that. most, teachers would be implementing after onjB year.' 
Of further significance, was the ^wealth of formative evaluation infor- 

tDatlon provided by^ interviews ihe areas of student -characteristics, 

*. , ' ' - ' » • 

materials, objectives, testing arid classroom 'managemeat. This additional 

information ide'ntlfied common practices and" the most frequenf. ways thai 

teachers implemented the program, prevailing problems, an^ithe typical ' 

organizational patterns supportin^jth^. innovation within the sc&ool.\ The 

informa.tion proved useful in identifying areas of need and allowed project 

targeted af program improvement . Th6" 



staff 90 make data-based deci'sions 



was ^specially valuable in 



its application to t^crpurposes' of 



formativi evaluation by pi^ctitionei's 



responaible for effective change ik schoa^s. The major' limitation cited 
for- the technique was ^ the cost to fcarry out'vali4 evaluation." 
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Appendix 1 



iterview •* 



0-11/llI-Vl 



Are >'ou currencly using 



NO 



Have you ever used it in the past? If so, when? Why did you scopi 
(if yes, go tq ^ then return) 



O/I-II 
I/II 

Acquiring 
Informacion 

lloovledge 

f 

AssMsiag 

Sharing 
Planning 



Final 
Qu^isciofi 
(Opcional) 



* Have you made a decision to use in the future? 
If so% when wiil you begin use? 

Cun you describe for roe as you see it? 

• Ar^ you currently looking for any information about 
^ ? What kinds? For what purposes?, 

Vbat da you sec as the strengths and weaknesses of 
in yox^situafon? , 

At this point in tiae, what kinds of questions'^e . 
. you asking about . ? Give examples if 

necessary. 

Do you ever talk with others and share infonnation 
•bout ^ What do you share? ^ 

What ate you planning with respect to ? Can 

you tell^me about any preparation or plans you have been* 
making for the use of ? 

Can you suimnarise for me w^ere you see yourself ri^ht now 
in relation, to the use of ? 



t<hen you assess 
nnd weaknesses? 




PAST USERS* 



w you organized your use of 



Can you describe for a _ 
you found, wluic Its eff/cts/ appeared to be on students? 



^ , what problems 



at this point timt, what do you^iee as the strengths 



(Return to other nonuse questions.) 
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Opon-enJcJ 



Assessing/ 

Kxiovlftdg« 



Acquirlqg 
InfoiWcionr 

LoU V 

r 



►Sharing 
Assessing 



YES 

Vlen^e Uc«crtbe fr»r mc how yon use 
qu^scions co gee cpnf iguracions. ) 



(Ask <<^flr:iont 



^ What do^you sae as ch€ screngchs .and weaknesses of 



your #ituscion? (Have you a^e'ai^raccem^rn^ranvching aboi? 
wcnknMScs?. Pjobc chose chcy mentioned specif icall/r) 

Arc you currendy IdokiiTg for Iny info 



)rmacion abouc 



_? Do you oieec 



III/IVA/IVB 



Planning/ 
Status 
Reporting 



What Idnd? For what . purposes?^ 

Do you work with others in your use of . ^ 

on a regular basis? Jipve you made any changes in yqur use of 
_ ."based 9a this coordination? (if yes, go to *) 

Dpjrou ever talk .with others about _^ ? ;;hat do you tell 

(Have you considered any alternatives or different ways of doing 
thln^ with the program?) Are you doing any evaluating, either 
fonaaUy or informally, that woyld affect your use^of 

you received any feedback from students that would affect 

the way you're using _? what have you done with the 

xnrormation you got? 

Have you made any changes recently in how you use 



AllI-V/VI 



What? Why?, How recently.? Are you considering making any 
changes? * 

As you look ahead to later this year, what plans do you have 

in relation to your use of ? 



Are you tonsiiering'or planning to make 'major modif icacions 
or replace ac this time? 



FRir 



♦ »LoU V Probes , 

How do you work together?^ What things do you shnre with (inch other? 
How frequently? 

What do you see a's the effects of this collAboration? 

Are you looking for any particular ki;td of information in relation to this 
collaboration? 

5. Do you talk with others about your collaboration? If so, what do you share 
with them? 

6. Have you done any formdl ot^nfonnal evaluation of how your collaboration is 
. working? 1 ' 

7.. What plans do you have for t)xi% effort in the fuGare? 

If you hav^ enough evidence to place the person at an LpU V, 
go ton 

If you do not' think tlie person is'an («cU V« 
go co«^ 
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WHY LoU INTERVIEWS WORK. 

1. Unique product 

. " . evaluation independence 
. confidentiality ^ 
, can't get from surveys 

.. • ■ • 

2. Documents PROGRAM sta^hjs^^ 

3. Implementation criteria^^Better definition 
^. Unanticipated outcomes 
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